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. Abstract 



Even when a system is proven to be correct with respect to a specification, there is still a question 
of how complete the specification is, and whether it really covers all the behaviors of the system. Cov- 
erage metrics attempt to check which parts of a system are actually relevant for the verification process 
C ■ \ to succeed. Recent work on coverage in model checking suggests several coverage metrics and algo- 

rithms for finding parts of the system that are not covered by the specification. The work has already 
proven to be effective in practice, detecting design errors that escape early verification efforts in indus- 
trial settings. In this paper, we relate a formal definition of causality given in |Halpern and Pearl 200 1| 
to coverage. We show that it gives significant insight into unresolved issues regarding the definition of 
coverage and leads to potentially useful extensions of coverage. In particular, we introduce the notion 
q | of responsibility, which assigns to components of a system a quantitative measure of their relevance to 

the satisfaction of the specification. 

> 

^sD . 1 Introduction 
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In model checking, we verify the correctness of a finite-state system with respect to a desired behavior by 
checking whether a labeled state-transition graph that models the system satisfies a specification of this 
behavior [Clark e, Grumberg, andPeled 1999| . An important feature of model-checking tools is their abil- 
ity to provide, along with a negative answer to the correctness query, a counterexample to the satisfaction 
of the specification in the system. These counterexamples can be essential in detecting subtle errors in 
complex designs [Clarke, Grumberg, McMi llan, and Zhao 1995| . On the other hand, when the answer to 
the correctness query is positive, most model-checking tools terminate with no further information to the 
user. Since a positive answer means that the system is correct with respect to the specification, this may 



seem to be reasonable at first glance. 

In the last few years, however, there has been growing awareness that further analysis may be nec- 
essary even if a model checker reports that a specification is satisfied by a given system. The concern 
is that the satisfiability may be due to an error in the specification of the desired behavior or the mod- 
elling of the system, rather than being due to the correctness of the system. Two main lines of research 
have focused on techniques for checking such errors. One approach involves vacuity detection, that is, 
checking whether the specification is satisfied for vacuous reasons in the model [ Be atty and Bryant 1994| 
|Beer, Ben-David, Eisner, and Rodeh 1997tlKurshan 19981 Kupferm an and Vardi 1999|IPurandare and Somenzi 2 0021. 



One particularly trivial reason for vacuity is that the specification is valid; perhaps more interesting are 
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cases of antecedent failure or valid/unsatisfiable constraints in the system. For example, the branching- 
time specification AG(req — > AFgrant) (every request is eventually followed by a grant on every path) is 
satisfied vacuously in a system where requests are never sent. A specification that is satisfied vacuously is 
likely to point to some problems in the modelling of the system or its desired behavior. 

A second approach, which is more the focus of this paper, uses what is called coverage estimation. 
Initially, coverage estimation was used in simulation-based verification techniques, where coverage met- 
rics are used in order to reveal states that were not visited during the testing procedure (i.e, not "covered" 
by this procedure); see [Dill 1998 , IPeled 200 1 ] for surveys. In the context of model checking, this intu- 
ition has to be modified, as the process of model checking may visit all the states of the system regard- 
less of their relevance to the satisfaction of the specification. Intuitively, a component or a state is cov- 
ered by a specification tp if changing this component falsifies ijj (see [ Hoskote, Ka m, Ho, and Zhao 1999| 
Chockler, Ku pferman , and Vardi 20 01 ]). For example, if a specification requires that AG(req — > AFgrant) 
holds at an initial state, and there is a path in which req holds only in one state, followed by two states 
both satisfying grant, then neither of these two states is covered by the specification (changing the truth of 
grant in either one does not render the specification untrue). On the other hand, if there is only one state 
on the path in which grant holds, then that state is covered by the specification. The intuition is that the 
presence of many uncovered states suggests that either the specification the user really desires has more 
requirements than those explicitly written (for example, perhaps the specification should really require a 
correspondence between the number of requests and grants), or that the system contains redundancies, and 
can perhaps be simplified (for example, perhaps there should be only a single grant on the path). This ap- 
proach has already proven to be effective in practice, detecting design errors that escape early verification 
efforts in industrial settings [Hoskote , Kam, Ho, and Zhao 1999 1. 

Roughly speaking, coverage considers the question of what causes the system to satisfy the specifica- 
tion. The philosophy literature has long been struggling with the problem of defining what it means for one 
event to cause another. In this paper, we relate a formal definition of causality given in [Halpern and Pearl 2001 1 
to coverage. We show that it gives significant insight into unresolved issues regarding the definition of cov- 
erage, and leads to potentially useful extensions of coverage. 



The definition of causality used in [Halpern and Pearl 2001 1, like other definitions of causality in the 
philosophy literature going back to Hume [Hume 1939 1, is based on counter/actual dependence. Essen- 
tially, event A is a cause of event B if, had A not happened (this is the counterfactual condition, since A 
did in fact happen) then B would not have happened. Unfortunately, this definition does not capture all the 
subtleties involved with causality. (If it did, there would be far fewer papers in the philosophy literature!) 
For example, suppose that Suzy and Billy both pick up rocks and throw them at a bottle. Suzy's rock gets 
there first, shattering the bottle. Since both throws are perfectly accurate, Billy's would have shattered the 
bottle had it not been preempted by Suzy's throw. (This story is taken from [Hall 2003 1.) Thus, according 
to the counterfactual condition, Suzy's throw is not a cause for shaterring the bottle. This problem is dealt 
with in [Halpern and Pearl 2001 1 by, roughly speaking, taking A to be a cause of B if B counterf actually 
depends on A under some contingency. For example, Suzy's throw is a cause of the bottle shattering 
because the bottle shattering counterfactually depends on Suzy's throw, under the contingency that Billy 
doesn't throw. It may seem that this solves one problem only to create another. While this allows Suzy's 
throw to be a cause of the bottle shattering, it also seems to allow Billy's throw to be a cause too. 

Why do most people think that Suzy's throw is a cause and Billy's is not? Clearly, it is because Suzy's 
throw hit first. As is shown in [Halpern and Pearl 2001| , in a naive model that does not take into account 
who hit first, both Suzy's throw and Billy's throw are in fact causes. But in a more sophisticated model 
that can talk about the fact that Suzy's throw came first, Suzy's throw is a cause, but Billy's is not. One 
moral of this example is that, according to the | Halp ern and Pearl 2001 1 definitions, whether or not A is a 



cause of B depends in part on the model used. Event A can be the cause of event B in one model and not 
in another. 

What is the connection of all this to coverage? First, note that the main definitions of coverage in 
the literature are inspired by counterfactual dependence. Indeed, a state s is p-covered by the specifi- 
cation ip if, had the value of the atomic proposition p been different in state s, then ip would not have 
been true. The initial definition of coverage [ Hoskot eTKam, Ho, and Zhao 1999| and its generalization in 
IChockler, Kupferman, and Vardi 2001 1 can be understood in terms of causality. The variant definition of 
coverage used in the algorithm proposed in [Hoskote , Kam, Ho, and Zhao 1999| , which the authors say is 
"less formal but meets our intuitions better", can also be described as an instance of causality. In fact, the 
variant definition can be captured using ideas similar to those needed to deal with the Suzy-Billy story. 
For example, the distinction in [ Hoskot e, Kam, Ho, and Zhao 1999 1 between the first position in which an 
eventuality is satisfied and later positions in which the eventuality is satisfied is similar to the distinction 
between Suzy, whose rock gets to the bottle first, and Billy, whose rock gets there later. 

Thinking in terms of causality has other advantages. In particular, using an extension of causality 
called responsibility, introduced in a companion paper [Chockler and Halpern 2003], we can do a more 
fine-grained analysis of coverage. To understand this issue, let us return to Suzy and Billy, and consider 
a scenario in which their rocks get to the bottle at exactly the same time. If we identify causality with 
counterfactual dependence, then both Suzy and Billy can claim that her or his rock does not cause the 
bottle to shatter. On the other hand, according to the definition in | Halpern and Pearl 2001 1 , both Suzy 
and Billy are causes of the bottle shattering (for example, the bottle shattering depends counterfactually 
on Suzy's throw if Billy does not throw). We would like to say that Suzy and Billy each have some 
responsibility for the bottle being shattered, but Suzy, for example, is less responsible than she would be in 
a scenario in which she is the only one that throws a rock. And if, instead of just Suzy and Billy, there are 
100 children all throwing rocks at the bottle, hitting it simultaneously, we would like to say that each child 
is less responsible for the bottle being shattered than in the case of Suzy and Billy and their two rocks. 

Going back to coverage, note that a state either covers a specification, or it doesn't. This all-or-nothing 
property seems to miss out on an important intuition. Consider for example the specification EXp. There 
seems to be a qualitative difference between a system where the initial state has 100 successors satisfying 
p and one where there are only two successors satisfying p. Although, in both cases, no state is p-covered 
by the specification, intuitively, the states that satisfy p play a more important role in the case where there 
are only two of them than in the case where there are 100 of them. That is, each of the two successors is 
more responsible for the satisfaction of EXp than each of the 100 successors. 

According to the definition in [Chockler and Halpern 2003 1, the degree of responsibility of a state s 
for a specification ip is a number between and 1 . A state s is covered by specification ip iff its degree 
of responsibility for ip is 1 ; the value of s is a cause of ip being true if the degree of responsibility of s 
for ip is positive. A degree of responsibility says intuitively that s plays no role in making ip true; a 
degree of responsibility strictly between and 1 says that s plays some role in making ip true, even if s by 
itself failing will not make ip false. For example, if the specification is EXp and the initial state has two 
successors where p is true, then the degree of responsibility of each one for EXp is 1/2; if there are one 
hundred successors where p is true, then the degree of responsibility of each one is 1/100. 

The issue of responsibility becomes particularly significant when one considers that an important rea- 
son that a state might be uncovered is due to fault tolerance. Here, one checks the ability of the system to 
cope with unexpected hardware or software faults, such as power failure, a link failure, a Byzantine fault, 
etc. [Lynch 1996 1. It is often the case that fault tolerance is achieved by duplication, so that if one compo- 
nent fails, another can take over. Accordingly, in this analysis, redundancies in the system are welcome: a 
state that is covered represents a single point of failure; if there is some physical problem or software 



problem that involves this state, then the specification will not be satisfied. To increase fault tolerance, 
we want states to be uncovered. On the other hand, we still want states to somehow "carry their weight". 
Thus, from the point of view of fault tolerance, while having a degree of responsibility of 1 is not good, 
since it means a single point of failure, a degree of responsibility of 1/100 implies perhaps unnecessary 
redundancy. 



2 Definitions and Notations 



In this section, we review the definitions of causality and responsibility from [Halpern and Pearl 2001 1 
and [Chockler and Halpern 2003]. As we argue below, models in formal verification are binary, thus 
we only present the significantly simpler versions of causality and responsibility for binary models (see 
I Eit er and Lukasiewicz 200 2b | for the simplification of the definition of causality for the binary case). We 
also omit several other aspects of the general definition including the division of variables to exogenous 
and endogenous. Readers interested in the general framework of causality are refered to Appendix |A] 
We also present the definitions of causality and responsibility for Boolean circuits and argue that binary 
recursive causal models are equivalent to Boolean circuits. We use Boolean circuits in our algorithms for 
computing responsibility in model checking and we justify this choice in Section 1331 



2.1 Binary causal models 

Definition 2.1 (Binary causal model) A binary causal model M is a tuple (V, T), where V is the set of 

boolean variables and T associates with every variable X € V a function Fx that describes how the 
value of X is determined by the values of all other variables in V. A context u is a legal setting for the 
variables in V. 



A causal model M is conveniently described by a causal network, which is a graph with nodes corre- 
sponding to the variables in V and an edge from a node labeled X to one labeled Y if Fy depends on the 
value of X. We restrict our attention to what are called recursive models. These are ones whose associated 
causal network is a directed acyclic graph. 

A causal formula tp is a boolean formula over the set of variables V. A causal formula ip is true or 
false in a causal model given a context. We write (M, u) \= ip if ip is true in M given a context u. We 
write (M, u) \= [Y <— y\ (X = x) if the variable X has value x in the model M given the context u and 
the assignment y to the variables in the set Y C V. 

With these definitions in hand, we can give the definition of cause from [Halp ern and Pearl 200 1| 
Eiter and Lukasiewicz 2002b | . 



Definition 2.2 (Cause) We say that X = x is a cause of ip in (M, u) if the following conditions hold: 
ACl. (M, u) (= (X = x) A ip. 

AC2. There exist a subset W ofV with X £ W and some setting (x', w') of the variables in (X, W) such 
that the following two conditions hold: 

(a) (M,u) \= [X <— x',W <— w'j^ip. That is, changing (X,W) from (x,w) to (x',w') changes 
ipfrom true to false. 



(b) (M, u) \= [X <— x, W <— r/jaf jj, setting W to w' should have no effect on <p as long 

as X has the value x. 



The definition of responsibility refines the "all-or-nothing" concept of causality by measuring the de- 
gree of responsibility of X = x in the truth value of tp in (M, u). The definition of responsibility is due to 
|Chockler and Halpern 2003 | , and we give here only the simpler definition for binary models. 



Definition 2.3 (Responsibility) The degree of responsibility of X = x for the value of p in (M, u), 

denoted dr((M, u),X = x, tp), is 1/(\W\ + 1), where W C V is the smallest set of variables that satisfies 
the condition AC2 in Definition \2.2\ 



Thus, the degree of responsibility measures the minimal number of changes that have to be made in u 
in order to falsify <p. If X = x is not a cause of p in (M, u), then the minimal set W in Definition 12.31 
is taken to have cardinality oo, and thus the degree of responsibility of X = x is 0. If ip counterfactually 
depends on X = x, then its degree of responsibility is 1. In other cases the degree of responsibility is 
strictly between and 1. Note that X = x is a cause of p iff the degree of responsibility of X = x for the 
value of ip is greater than 0. 



2.2 Causality and responsibility in Boolean circuits 

In this section, we consider an important setting in which to consider causality and responsibility: Boolean 
circuits. A Boolean circuit is just a representation of a propositional formula, where the leaves represent 
atomic propositions and the interior nodes represent the Boolean operations -i, A, and V. Given an assign- 
ment of values to the leaves, the value of the root is the value of the formula. Without loss of generality, we 
assume that propositional formulas are in positive normal form, so that negation is applied only to atomic 
propositions. (Converting a formula to an equivalent formula in positive normal form at most doubles the 
length of the formula.) Thus, in the Boolean circuit, negations occur only at the level above the leaves. We 
also assume without loss of generality that all A and V gates in a Boolean circuit are binary. 

Let g : {0, l} n — > {0, 1} be a Boolean function on n variables, and let C be a Boolean circuit that 
computes g. As usual, we say that a circuit C is monotone if it has no negation gates. We denote by X the 
set of variables of C. A truth assignment / to the set X is a function / : X — > {1, 0}. The value of a gate w 
of C under an assignment / is defined as the value of the function of this gate under the same assignment. 
Thus, we can extend the domain of / to all gates of the circuit. For an assignment / and a variable X, we 
denote by fx the truth assignment that differs from / in the value of X. Formally, fx(Y) = f(Y) for all 
Y ^ X, and fx{X) = ->f(X). Similarly, for a set Z C X, fg is the truth assignment that differs from / 
in the values of variables in Z. 

It is easy to see that Boolean circuits are a special case of binary causal models, where each gate of 
the circuit is a variable of the model, and values of inner gates are computed based on the values of the 
inputs to the circuit and the Boolean functions of the gates. A context u is a setting to the input variables 
of the circuit. For the ease of presentation, we explicitly define the notion of criticality in Boolean circuits, 
which captures the notion of counter-factual causal dependence. 

Definition 2.4 Consider a Boolean circuit C over the set X of variables, an assignment f, a variable 
X £ X, and a gate w of C. We say that X is critical for w under f if fx(w) = -*f(w). 



If a variable X is critical for the output gate of a circuit C, changing the value X alone causes a change 
in the value of C. If X is not critical, changing its value alone does not affect the value of C. However, 
it might be the case that changing the value of X together with several other variables causes a change in 
the value of C. Fortunately, the definitions of cause and responsibility can be easily re-written for Boolean 
circuits, where the only causal formulas we consider are the formulas of the gates. 

Definition 2.5 Consider a Boolean circuit C over the set X of variables, an assignment f, a variable 
X S X, and a gate w ofC. A (possibly empty) set Z C X \ {X} makes X critical for w if fg{w) = f(w) 
and X is critical for w under fg. (The value of) X is a cause of (the value of) w if there is some Z that 
makes X critical for w. 

Similarly, we can re-write the definition of responsibility for Boolean circuits in the following way. 

Definition 2.6 (Degree of Responsibility) Consider a Boolean circuit C over the set X of variables, an 
assignment f, a variable X E X, and a gate w ofC. The degree of responsibility of (the value of) X for 
(the value of) w under f, denoted dr(C, X, w, f), is 1/(1 + \Z\), where Z C X \ {X} is a set of variables 
of minimal size that makes X critical for w under f. 

Thus, dr(C, X, w, f) measures the minimal number of changes that have to be made in / in order to 
make X critical for w. If no subset Z C X \ {X} makes X critical for w under /, then the minimal set 
Z in Definition 12.61 is taken to have cardinality oo, and thus the degree of responsibility of X is 0. If X 
is critical for w under /, then its degree of responsibility is 1. In other cases the degree of responsibility 
is strictly between and 1. We denote by dr(C,X, f) the degree of responsibility of X for the value 
of the output gate of C. For example, if / is the assignment that gives all variables the value 1, then 
dr(X\ V X2,X\, f) = 1/2, while cir(Vi£i Xi,X±, f) = 1/100. For another example, consider a circuit 
C = (X A Y) V (X A Z) V (Y A Z) V (X A U). That is, either two out of three variables X, Y, and Z 
should be assigned 1, or X and U should be assigned 1 in order for C to have the value 1. Consider an 
assignment f\ that assigns all variables the value 1. Then, dr(C, X, f\) = 1/3, since changing the value 
of two out of three variables Y, Z, and U does not affect the value of C, but changing the value of two out 
of three variables Y, Z, and U together with X falsifies C. Now consider an assignment f% that assigns Y, 
Z, and U the value 1, and X the value 0. Clearly, changing the value of X from to 1 cannot falsify C, 
thus dr(C, X, $2) = 0. Finally, consider an assignment /3 that assigns X and Y the value 1, and Z and U 
the value 0. In this case, changing the value of X alone falsifies C, so dr(C, X, fs) = 1. 

Remark 2.7 We note that while we define the degree of responsibility for a specific circuit, in fact its 
value depends solely on the Boolean function that is computed by the circuit and is insensitive to the 
circuit structure. Thus, degree of responsibility is a semantic notion, not a syntactic one. □ 

3 Coverage, Causality, and Responsibility in Model Checking 

In this section we show how thinking in terms of causality and responsibility is useful in the study of cov- 
erage. In Section lTTI we show that the most common definition of coverage in model checking conforms to 
the definition of counter-factual causality and demonstrate how the coverage information can be enhanced 
by the degrees of responsibility of uncovered states. In Section l3^2l we discuss other definitions of coverage 
that arise in the literature and in practice and describe how they fit into the framework of causality. 



3.1 Coverage in the framework of causality 



The following definition of coverage is perhaps the most natural one. It arises from the study of mu- 
tant coverage in simulation-based verification [Mill o~ipton, and Say ward 1978} iMillo and Qffutt 199T1 
lAmmann and Black 20 01 1, and is adopted in [Hoskot e, Kam, Ho, and Zhao 1999|[Chockler, Kupferman, and Vardi 2 001 ; 
Chockler, Kupferman, Kurshan, and Vardi 200T]|Chockler and Kupferman 2002[lamd M. Purandare and Somenzi 200311 . 
For a Kripke structure K, an atomic proposition q, and a state w, we denote by K w>q the Kripke structure 
obtained from K by flipping the value of q in w. Similarly, for a set of states Z, is the Kripke 
structure obtained from K by flipping the value of q in all states in Z. 

Definition 3.1 (Coverage) Consider a Kripke structure K, a specification tp that is satisfied in K, and an 
atomic proposition q € AP. A state w of K is g-covered by tp if K w q does not satisfy tp. 

It is easy to see that coverage corresponds to the simple counterfactual-dependence approach to causal- 
ity. Indeed, a state w of K is q-covered by ip if tp holds in K and if q had other value in w, then p would not 
have been true in K. The following example illustrates the notion of coverage and shows that the counter- 
factual approach to coverage misses some important insights in how the system satisfies the specification. 
Let K be a Kripke structure presented in Figure [J and let p = AG(req — > AF grant). It is easy to see 
that K satisfies tp. State wj is grant-covered by (p. On the other hand, states W2, w^, W4, and W5 are not 
grant-covered, as flipping the value of grant in one of them does not falsify tp in K. Note that while the 
value of grant in states 1V2, W3, and plays a role in the satisfaction of ip in K, the value of grant in w§ 
does not. One way to capture this distinction is by using causality rather than coverage. 

req grant 




Figure 1: States W2,w^, and are not covered by AG {req —>■ AF grant), but have degree of responsi- 
bility 1 /3 for its satisfaction. 

Definition 3.2 Consider a Kripke structure K, a specification ip that is satisfied in K, and an atomic 
proposition q S AP. A state w is a cause of <p in K with respect to q if there exists a (possibly empty) 
subset of states Y ofK such that flipping the value of q in Y does not falsify <p in K, and flipping the value 
of q in both w and Y falsifies p> in K. 

In Figure we describe a Kripke structure K in which the states W2, W3, w$, and wj are causes of 
AG(req — > AF grant) in K with respect to grant, while w$ is not a cause. This reflects the fact that while 
the value of grant is critical for the satisfaction of 99 only in the state wj, in states W2, w$, and the value 
of grant also has some effect on the value of tp in K. It does not, however, give us a quantative measure 
of this effect. Such a quantative measure is provided using the analogue of responsibility in the context of 
model checking. 



Definition 3.3 Consider a Kripke structure K, a specification p that is satisfied in K, and an atomic 
proposition q € AP. The degree of (/-responsibility of a state w for tp is 1/(\Z\ + 1), where Z is a subset 
of states of K of minimal size such that Kg satisfies p and w is q-covered by p in Kg . 

In the Kripke structure described in Figure [Q states u>2, W3, and have degree of responsibility 1/3 
for the satisfaction of AG(req — > AF grant), state u>5 has degree of responsibility 0, and state 107 has 
degree of responsibility 1, all with respect to the atomic proposition grant. 

Assigning to each state its degree of responsibility gives much more information than the yes/no an- 
swer of coverage. Coverage does not distinguish between states that are quite important for the satisfaction 
of the specification, even though not essential for it, and those that have very little influence on the satisfac- 
tion of the specification; responsibility can do this well. This is particularly relevant for specifications that 
implicitly involve disjunctions, such as formulas of the form EXip or EFip. Such specifications typically 
result in many uncovered states. Using responsibility gives a sense of how redundant some of these states 
really are. Moreover, as we observed in the introduction, any degree of redundancy in the system automat- 
ically leads to low coverage. On the other hand, for fault tolerance, we may actually want to require that 
no state has degree of state higher than, say, 1/3, that is, every state should be backed up at least twice. 

3.2 Other definitions of coverage 

In the previous section we showed that the definition of coverage used most often in the literature can be 
captured in the framework of causality. There is another definition for coverage given in [Hos kote, Kam, Ho, and Zhao 1999| 
that, while based on mutations, is sensitive to syntax. Thus, according to this definition, w may (/-cover 
(p but not (/-cover ip', although <p and <p' are semantically equivalent formulas. The justification for such 
syntactic dependencies is that the way a user chooses to write a specification carries some information. 
(Recall that the same issue arose in the case of Boolean circuits, although there we offered a different jus- 
tification for it.) The variant definition given in [Hoskot e, Kam, Ho, and Zhao 199 9 1 has two significant 
advantages: it leads to an easier computational problem, and it deals to some extent with the fact that very 
few states are covered by eventuality formulas, which implicitly involve disjunction. Moreover, according 
to [Ho skote, Kam, Ho, and Zhao 1999| , the definition "meets our intuitions better". 

Roughly speaking, the definition in [Hoskote, Kam, Ho, and Zhao 199 9 1 distinguishes between the 
first state where an eventuality is fulfilled and other states on the path. That is, if an eventuality p is first 
fulfilled in a state w in the original system and is no longer fulfilled in w in the mutant system obtained by 
flipping the value of q in some state v, then v is said to be (/-covered' by <p, even if ip is still satisfied in the 
mutant system. 

To define cover' precisely, a specification <p is transformed to a new specification trans q (p) that may 
include a fresh atomic proposition q', such that a state w is (/-covered' by <p in Kripke structure K iff w is q'- 
covered by trans q (<p) in the Kripke structure K' that extends K by defining q' to be true at exactly the same 
states as q. We do not give the full definition of trans q here (see |Hoskote, Kam, Ho, and Zhao 199 9 ]); 
however, to give the intuition, we show how it works for universal until formulas. Assuming that trans q 
has been recursively defined for <p and tp, let 

trans q (A(ipUip)) = A[trans q (p)Ut/j} A A[(p A -^ip)U tranship)}, 

where trans q (q) = q' , for some fresh atomic proposition q', and trans q {p) = p if p ^ q. Thus, for 
example, trans q (A(pUq)) = A(pUq) A (A(p A ^q)Uq'). It is not hard to see that if K satisfies A(pUq), 
then w (/-covers' A(pUq) iff w is the first state where q is true in some path in K. For example, let K be a 



structure that consists of a single path ir = wq, w%, W2, ■ ■ ., and assume that wq and w\ are the only states 
where p is true and that w\ and W2 are the only states where q is true. Then the specification tp = A(pUq) 
is satisfied in K and neither w± nor W2 is g-covered by p. Note that ip is fulfilled for the first time in w\ 
and that if we flip q in w\, w\ no longer fulfils the eventuality. Thus, w\ is g-covered' by ip. 

While the intuitiveness of this interpretation of coverage is debatable, it is interesting to see that this 
requirement can be represented in the framework of causality. Intuitively, the eventuality being fulfilled 
first in w\ is much like Suzy's rock hitting the bottle first. And just as in that example, the key to capturing 
the intuition is to add extra variables that describe where the eventuality is first fulfilled. Thus, we introduce 
two additional variables called Fl ("eventuality is first fulfilled in wi") and F2 ("eventuality is first fulfilled 
in W2"). This gives us the causal model described in Figure |2] 




Figure 2: The cause of ApUq being true in K is taken to be the first place where the eventuality is fulfilled. 

The definition of coverage for eventuality formulas in [Hoskote , Kam, Ho, and Zhao 1999| can be 
viewed as checking whether an eventuality formula is satisfied "in the same way" in the original model and 
the mutant model. Only a fragment of the universal subset of CTL is dealt with in [Hoskote , Kam, Ho, and Zhao 1999| , 
but this approach can be generalized to deal with other formulas that can be satisfied in several ways. For 
example, a specification ip = EXp is satisfied in a Kripke structure K if there exists at least one successor 
of the initial state wq labeled with p. If we want to check whether ip is satisfied in a mutant structure K' in 
the same way it is satisfied in the original system K, we introduce a new variable X w for each successor 
w of wq and we assign 1 to X w iff w is labeled with p. Then we replace model checking of ip in mutant 
systems by model checking of ip' = /\ w esucc(w ) lw> where l w is X w if X w = 1 and is ->X W otherwise. 
Clearly, a mutant system satisfies ip' iff the mutation does not affect the values of p in successors of the 
initial state. More generally, this idea of adding extra variables to check that certain features are preserved 
can be used to give a more fine-grained control over what coverage is checking for. 



3.3 Boolean circuits in model checking 

To motivate Boolean circuits in the context of model checking, we review the automata-theoretic approach 



to branching-time model checking, introduced in [Kupferman, Vardi, and Wolper 2000 [. We focus on the 



branching-time logic CTL. Formulas of CTL are built from a set AP of atomic propositions using the 
Boolean operators V and -1, the temporal operators X ("next") and U ("until"), and the path quantifiers 
E ("exists a path") and A ("for all paths"). Every temporal operator must be immediately preceded by 
a path quantifier. The semantics of temporal logic formulas is defined with respect to Kripke structures, 
which are labeled state-transition graphs; see I Emer son 1990 1 for details. Suppose that we want to check 
whether a specification ip written in branching-time temporal logic holds for a system described by a 
Kripke structure K. We assume that K has a special initial state denoted w: m . Checking if K satisfies tp 
amounts to checking if the model with root Wi n obtained by "unwinding" K satisfies tp. 

In the automata-theoretic approach, we transform p to an alternating tree automaton A v that accepts 
exactly the models of tp. Checking if K satisfies p is then reduced to checking the nonemptiness of the 



product Ak,<p of K and A v (where we identify K with the automaton that accepts just K). When <p> 
is a CTL formula, the automaton A v is linear in the length of p; thus, the product automaton is of size 
0(\K\ ■ \(p\). 

Let W be the set of states in K and let AP be the set of atomic propositions appearing in ip. The 
product automaton Ak& can be viewed as a graph Gk,^- The interior nodes of Gk,v are pairs (w,ip), 
where w E W and ip is a subformula of p that is not an atomic proposition. The root of Gk,^ is the 
vertex {wi n , p). The leaves of Gk, v are pairs (w,p) or (w, -ip), where w £ W and p 6 j4P. As shown in 
| Chockler, Kupfer man, and Vardi 2001] , we can assume that each interior node (w, ip) has two successors, 
and is classified according to the type of ip as an OR-node or an AND-node. Each leaf (w,p) or (w, ->p) 
has a value, 1 or 0, depending on whether p is in the label of state w in the model K. The graph has at 
most 2 • \AP\ ■ \ W\ leaves. 

We would like to view the graph Gk,<p as a Boolean circuit. To do this, we first replace each node 
labeled (w, ->p) by a NOT-node, and add an edge from the leaf (w,p) to the NOT-node. Clearly this does 
not increase the size of the graph. The only thing that now prevents Gx,ip from being a Boolean circuit 
is that it may have cycles. However, as shown in [Kupferman, Vardi , and Wolper 2000| , each cycle can 
be "collapsed" into one node with many successors; this node can then be replaced by a tree, where each 
node has two successors. The size of the resulting graph is still 0(|K| • \<p\). Model checking is equivalent 
to finding the value of the root of Gx,ip given the values of the leaves. That is, model checking reduces to 
evaluating a Boolean circuit. The following result is straightforward, given the definitions. 

Proposition 3.4 Consider a Kripke structure K, a specification ip, and an atomic proposition q. The 
following are equivalent: 

(a) the degree of q-responsibility of w for p is 1/k; 

(b) the node (w, q) has degree of responsibility 1/k for (wi n , p) in the Boolean circuit corresponding 
to K and <p; 

(c) X w>q has degree of responsibility 1/k for the output in the causal model corresponding to K and p. 

It is almost immediate from Proposition 13.41 that w is g-covered by tp in the Kripke structure K iff (w, q) 
is critical (i.e., has degree of responsibility 1) for the value of (wi n , p) in the Boolean circuit iff X WjQ has 
degree of responsibility 1 for the value of the output in the causal model. 

4 Computing the Degree of Responsibility in Binary Causal Models 

In this section we examine the complexity of computing the degree of responsibility. We start with the 
complexity result for the general case of binary causal models. Then we discuss several special cases for 
which the complexity of computing responsibility is much lower and is feasible for practical applications. 

4.1 The general case 

For a complexity class A, FP A l l °s n ] consists of all functions that can be computed by a polynomial- 
time Turing machine with an oracle for a problem in A, which on input x asks a total of 0(log \ x\) 
queries (cf. [ Papadimitriou 1984 1). Eiter and Lukasiewicz lEiter and Lukasiewicz 2002all show that test- 
ing causality is Sg'-complete; in |Chockler and Halpern 2003| , it is shown that the problem of computing 



responsibility is FP S 2 P°s«] -complete for general causal models. Eiter and Lukasiewicz showed that in 
binary causal models, computing causality is NP-complete. Since the causal model corresponding to a 
Boolean circuit is binary, computing causality is NP-complete in Boolean circuits. We show that com- 
puting the degree of responsibility is FP NP l logn l -complete in binary causal models. We actually prove the 
FP NP t logn ] -completeness first for Boolean circuits. Then we show that a slight extension of our argument 
can be used to prove the same complexity result for all binary causal models. 

Formally, the problem RESP-CIRCUIT is defined as follows: given a circuit C over the set of variables 
X, a variable lei, and a truth assignment /, compute dr(C, X, /). We prove the following theorem. 

Theorem 4.1 RESP-CIRCUIT is FP^ l °^ -complete. 

The proofs of Theorem l4.1l and its easy extension below can be found in Appendix iBl 

Theorem 4.2 Computing the degree of responsibility is FP NF ^ ogn ^ -complete in binary causal models. 

By Proposition 13.41 the upper bound in Theorem 14. 1 1 applies immediately to computing the degree of 
responsibilty of a state w for a formula tp. The lower bound also applies to model checking, since it is 
not hard to show that for every Boolean function / over the set of variables X and assignment x there 
exists a pair (K, tp) such that K is a Kripke structure, tp is a specification, and model checking of ip in K 
amounts to evaluating a circuit C that computes / under the assignment x. Indeed, let K be a single-state 
structure with a self-loop over the set X of atomic propositions, where the single state of K is labeled with 
X € X iff X is 1 under the assignment x. Let p be a prepositional formula over the set of variables X 
that computes the function /. Then the graph Gk,<p is a circuit that computes / and evaluating Gk,<p is 
equivalent to evaluating / under the assignment x. 

4.2 Tractable special cases 

Theorem 14.11 shows that there is little hope of finding a polynomial-time algorithm for computing the 
degree of responsibility for general circuits. The situation may not be so hopeless in practice. For one 
thing, we are typically not interested in the exact degree of responsibility of a node, but rather want a 
report of all the nodes that have low degree of responsibility. This is the analogue of getting a report of the 
nodes that are not covered, which is the goal of algorithms for coverage. As in the case of coverage, the 
existence of nodes that have a low degree of responsibility suggests either a problem with the specification 
or unnecessary redundancies in the system. 

Clearly, for any fixed k, the problem of deciding whether dr(C, X,w, f) > 1/k can be solved in 
time 0{\X\ k ) by the naive algorithm that simply checks whether X is critical for C under the assign- 
ment fg for all possible sets Z C X of size at most k — 1. The test itself can clearly be done in 
linear time. We believe that, as in the case of coverage, where the naive algorithm can be improved 
by an algorithm that exploits the fact that we check many small variants of the same Kripke structure 
[Chockler, Kupferma n, and Vardi 2001) , there are algorithms that are even more efficient. In any case, 
this shows that for values of k like 2 or 3, which are perhaps of most interest in practice, computing 
responsibility is quite feasible. 

There is also a natural restriction on circuits that allows a linear-time algorithm for responsibility. We 
say that a Boolean formula ip is read-once if each variable appears in tp only once. Clearly, a Boolean 
circuit for a read-once formula is a tree. While only a small fraction of specifications are read-once, 
every formula can be converted to a read-once formula simply by replacing every occurrence of an atomic 



proposition by a new atomic proposition. For example, tp = (p A q) V (p A r) can be converted to 
V'' = (po A 9) V (pi A r ) . Given an assignment for the original formula, there is a corresponding assignment 
for the converted formula that gives each instance of an atomic proposition the same truth value. While this 
does not change the truth value of the formula, it does change responsibility and causality. For example, 
under the assignment that gives every atomic proposition the value 1, p is critical for ip and thus has 
responsibility 1 for the value of tp, while under the corresponding assignment, po has responsibility only 
1/2 for ifi'. Similarly, p is not a cause of the value of p V ->p under the assignment that gives value 1 to p, 
but po is cause of the value of po V ->pi under the corresponding assignment. 

If we think of each occurrence of an atomic proposition as being "handled" by a different process, 
then as far as fault tolerance goes, the converted formula is actually a more reasonable model of the 
situation. The conversion models the fact that each occurrence of p in ip can then fail "independently". This 
observation shows exactly why different models may be appropriate to capture causality. Interestingly, 
this type of conversion is also used in vacuity detection in [Beer, B en-David, Eisner, and Rodeh 1997| 
Kupfer man and Vardi 1999|IPurandare and Somenzi 20021 . where each atomic proposition is assumed to 
have a single occurrence in the formula. 

In model checking, we can convert a Boolean circuit obtained from the product of a system K with 
a specification <p to a read-once tree by unwinding the circuit into a tree. This results in a degree of 
responsibility assigned to each occurrence of a pair (w, ip), and indeed each pair may occur several times. 
The way one should interpret the result is then different than the interpretation for the Boolean-circuit 
case and has the flavor of node coverage introduced in [Chockler, Kupferman, Kursh anT and Vardi 2001| . 
Essentially, in node coverage, one measures the effect of flipping the value of an atomic proposition in a 
single occurrence of a state in the infinite tree obtained by unwinding the system. 

The general problem of vacuity detection for branching-time specifications is co-NP-complete; the 
problem is polynomial for read-once formulas [Kupferman and Vardi 1999]. Considering read-once for- 
mulas also greatly simplifies computing the degree of responsibility. To prove this, we first need the 
following property of monotone Boolean circuits. 

Lemma 4.3 Given a monotone Boolean circuit C over the set X of variables, a variable X £ X, a gate 
w € C, and an assignment f, if f(w) 7^ f{X), then dr(C, X, w, f) = 0. 

Proof: Both functions A and V are monotone non-decreasing in both their variables, and thus also their 
composition is monotone non-decreasing in each one of the variables. Each gate of C is a composition of 
functions A, V over the set X of variables, thus all gates of C are monotone non-decreasing in each one 
of the variables of C. A gate w represents a function over the basis {A, V}. The assignment / assigns the 
variable X a value in {0, 1}, and f(w) is computed from the values assigned by / to all variables of C. We 
assume that f(X) ^ f{w). Without loss of generality, let f(X) = 1 and f(w) = 0. Assume by way of 
contradiction that dr(C, X, w, f) ^ 0. Then there exists a set Z C X \ {X} such that fg(w) = f(w) = 
and X is critical for w under /s. Thus, changing the value of X from 1 to changes the value of w from 
to 1. However, this contradicts the fact that w is monotone nondecreasing in X. 

The case where f(X) = and f(w) = 1 follows by a dual argument. □ 

Theorem 4.4 The problem of computing the degree of responsibility in read-once Boolean formulas can 
be solved in linear time. 

Proof: We describe a linear-time algorithm for computing the degree of responsibility for read-once 
Boolean formulas. Since we have assumed that formulas are given in positive normal form, we can as- 
sume that the trees that represent the formulas do not contain negation gates. (The leaves may be labeled 



with negations of atomic propositions instead.) This means that the circuits corresponding to read-once 
formulas can be viewed as monotone Boolean treess, to which Lemma l4~3l can be applied. 

Consider the following algorithm, which gets as in put a monotone Boolean tree T, an assignment /, 
and a variable X whose degree of responsibility for the value of T under the assignment / we want to 
compute. The algorithm starts from the variables and goes up the tree to the root. For each node w in the 
tree, the algorithm computes two values, size(T, X, w, f), which is the size of the minimal Z such that X 
is critical for w under fg, and the c(w, /), the size of the minimal Z such that Z C X and fg(w) ^ f(w). 
Note that size(T, X, w, f) = dr{T ^ x , w ,f) ~ 1 - 

For a leaf lx labeled with X, we have c(lx, f ) = 1 and size(T, X, lx, f) = 0, by Definition 12.61 For 
a leaf ly labeled with F/Iwe have c(ly, /) = 1 and size(T, X, ly, f ) = 0. Let w be a gate that is 
fed by gates u and v, and assume we have already computed size(T, X, y, f) and c(y, /), for y G {u, v}. 
Then size(T, X, w, f ) and c(w, f) are computed as follows. 

1. If size(T, X, u, f) = size(T, X, v, /) = oo, then size(T, X, w, f) = oo. 

2. If w is an A-gate and f(w) = f(u) = f(v) = 0, or if w is V-gate and f(w) = f(u) = f{v) = 1, 
then c(w) = c(u) + c(v) (because we have to change the values of both u and v in order to change 
the value of w), and the size of minimal Z is computed as follows. 

(a) If size(T, X, u,f)=i and size(T, X, v, f) = oo, then size(T, X,w,f) = i + c(v). 

(b) The case where size(T, X,u, f) < oo and size(T, X,v, f) < oo is impossible, since this 
would mean that X is a successor of both u and v, contradicting the tree structure of T. 

3. If w is an A-gate, f(w) = f(u) = and f(v) = 1, or if w is an V-gate, f(w) = f(u) = 1, and 
f(v) = 0, then c(w) = c(u), and the size of minimal Z is computed as follows. 

(a) If size(T, X, u,f)=i and size(T, X, v, f) = oo, then size(T, X, w, f) = i. 

(b) If size(T, X, v, /) = i and size(T, X, u, f ) = oo, then size(T, X, w, /) = oo by Lemma l431 

(c) The case where size(T, X, u, f ) = i and size(T, X, v, f ) = j is impossible by Lemma l4~3l 

4. If w is an A-gate and f(w) =/(«) = f(v) = 1, or if w is an V-gate and f(w) = f(u) = f(v) = 0, 
then c(w) = min(c(u),c(v)), and the size of minimal Z is computed as follows. 

(a) If size(T, X, u, f ) = i and size(T, X, v, /) = oo, then size(T, X, w, f) = i. 

(b) The case where size(T,X,u, f) < oo and size(T, X,v, f) < oo is impossible, since X 
cannot be a successor of both u and v in the tree T. 

Clearly we can compute the size(T,X,w, /) and c(w,f) in constant time (given the information 
that we already have at the time when we perform the computation). Moreover, because T is a tree, it 
is easy to check that size(T, X, w, f ) really is the size of the minimal Z such that X is critical for w 
under fg. As we observed earlier, the degree of responsibility of X for the value of node w under / is 
1/(1 + size(T, X, w, /)). Therefore, we proved the following proposition. □ 

5 Conclusion 

We have shown that it is useful to think of coverage estimation in terms of causality. This way of thinking 
about coverage estimation not only shows that a number of different definitions of coverage can be thought 



of as being defined by different models of causality, but also suggests how the notion of coverage might 
be extended, to take into account which features of satisfaction are important. The notion of responsibility 
also provides a useful generalization of coverage, that gives a more fine-grained analysis of the importance 
of a state for satisfying a specification. Our complexity results suggest that these notions can be usefully 
incorporated into current model-checking techniques. 
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A The General Framework of Causality 

In this section, we review the details of the definitions of causality and responsibility from jHal pern and Pearl 2001 1 
and [ Chockler and Halpern 2003 1 . 

A signature is a tuple S = (Li, V, TV), where Li is a finite set of exogenous variables, V is a set of 
endogenous variables, and the function 1Z : U U V — > V associates with every variable Y € Li U V a 
nonempty set TZiY) of possible values for Y from the range V. Intuitively, the exogenous variables are 
ones whose values are determined by factors outside the model, while the endogenous variables are ones 
whose values are ultimately determined by the exogenous variables. A causal model over signature S is 
a tuple M = (S,T), where T associates with every endogenous variable X € V a function Fx such 
that F x : (xueuK(U)) x (xYeV\{x}K(Y)) -> TZ(X). That is, F x describes how the value of the 
endogenous variable X is determined by the values of all other variables in Li U V. If the range V contains 
only two values, we say that M is a binary causal model. 

We can describe (some salient features of) a causal model M using a causal network. This is a graph 
with nodes corresponding to the random variables in V and an edge from a node labeled X to one labeled 
Y if Fy depends on the value of X. Intuitively, variables can have a causal effect only on their descendants 
in the causal network; if Y is not a descendant of X, then a change in the value of X has no affect on 
the value of Y. For ease of exposition, we restrict attention to what are called recursive models. These 
are ones whose associated causal network is a directed acyclic graph (that is, a graph that has no cycle of 
edges). It should be clear that if M is a recursive causal model, then there is always a unique solution to 
the equations in M, given a context, that is, a setting u for the variables in Li. 

The equations determined by {Fx ■ X £ V} can be thought of as representing processes (or mech- 
anisms) by which values are assigned to variables. For example, if Fx(Y, Z,U) = Y + U (which we 
usually write as X = Y + U), then if Y = 3 and U = 2, then X = 5, regardless of how Z is set. This 
equation also gives counterfactual information. It says that, in the context U — 4, if Y were 4, then X 
would be u + 4, regardless of what value X, Y, and Z actually take in the real world. 

While the equations for a given problem are typically obvious, the choice of variables may not be. For 
example, consider the rock-throwing example from the introduction. In this case, a naive model might 
have an exogenous variable U that encapsulates whatever background factors cause Suzy and Billy to 
decide to throw the rock (the details of U do not matter, since we are interested only in the context where 
J7's value is such that both Suzy and Billy throw), a variable ST for Suzy throws (ST = 1 if Suzy throws, 
and ST = if she doesn't), a variable BT for Billy throws, and a variable BS for bottle shatters. In the 
naive model, BS is 1 if one of ST and BJis 1. 



Figure 3: The rock-throwing example. 



This causal model does not distinguish between Suzy and Billy's rocks hitting the bottle simultane- 
ously and Suzy's rock hitting first. A more sophisticated model is the one that takes into account the fact 
that Suzy throws first. It might also include variables SH and BH, for Suzy's rock hits the bottle and Billy's 
rock hits the bottle. Clearly BS is 1 iff one of BH and BT is 1. However, now, SH is 1 if ST is 1, and 
BH = 1 if BT = 1 and SH = 0. Thus, Billy's throw hits if Billy throws and Suzy's rock doesn't hit. This 
model is described by the following graph, where there is an arrow from variable X to variable Y if the 
value of Y depends on the value of X. (The graph ignores the exogenous variable U, since it plays no 
role.) 

Given a causal model M = (S, J 7 ), a (possibly empty) vector X of variables in V, and vectors x 
and u of values for the variables in X and hi, respectively, we can define a new causal model denoted 
M x^x over ±e si g nature = (U, V - X, K\ v _%). Formally, = (S^F*^ 3 ), where Fy*~ 3 

is obtained from Fy by setting the values of the variables in X to x. Intuitively, this is the causal model 
that results when the variables in X are set to x by some external action that affects only the variables in 
X; we do not model the action or its causes explicitly. For example, if M is the more sophisticated model 
for the rock-throwing example, then Mst<-o is the model where Suzy doesn't throw. 

Given a signature S = (Li, V, TV), a formula of the form X = x, for X G V and x G 1Z(X), is called 
a primitive event. A basic causal formula is one of the form \Y\ <— yi, . . . , Yj. <— yk]<p, where ip is a 
Boolean combination of primitive events; Y\, . . . , Yf. are distinct variables in V; and yj € 1Z(Yi). Such 
a formula is abbreviated as [Y <— y\tp. The special case where k = is abbreviated as ip. Intuitively, 
[Yi yi, . . . , Yfe <— yk]<p says that ip holds in the counterfactual world that would arise if Yi is set to yi, 
i = 1, . . . , k. A causal formula is a Boolean combination of basic causal formulas. 

A causal formula ip is true or false in a causal model, given a context. We write (M, u) \= p if ip is 
true in causal model M given context u. (M, u) (= [Y <— y\(X = x) if the variable X has value x in 
the unique (since we are dealing with recursive models) solution to the equations in in context u 

(that is, the unique vector of values for the exogenous variables that simultaneously satisfies all equations 

F z y , Z G V — Y, with the variables in U set to u). We extend the definition to arbitrary causal formulas 
in the obvious way. 

With these definitions in hand, we can give the definition of cause from [Halper n and Pearl 2001 1. 
Definition A.l We say that X = x is a cause of ip in (M, u) if the following three conditions hold: 
ACl. (M, u) H (X = x) A ip. 

AC2. There exist a partition (Z,W) of V with X C Z and some setting (x',w') of the variables in 
(X, W) such that if (M, u) \= Z = z* for Z G Z, then 

(a) (M, u) \= [X <— x', W <— w']-«p. That is, changing (X, W) from (x, w) to (x" , w') changes 
tpfrom true to false. 



(b) (M,u) \= [X <— x, W <— w',Z' <— z*]y?/or aZZ subsets Z' of Z. That is, setting W to 
w 1 should have no effect on ip as long as X has the value x, even if all the variables in an 
arbitrary subset of Z are set to their original values in the context u. 

AC3. (X = x) is minimal, that is, no subset of X satisfies AC2. 

AC1 just says that A cannot be a cause of B unless both A and B are true, while AC3 is a minimality 
condition to prevent, for example, Suzy throwing the rock and sneezing from being a cause of the bottle 
shattering. Eiter and Lukasiewicz I Eite r and Lukasiewicz 200 2b I showed that one consequence of AC3 is 
that causes can always be taken to be single conjuncts. The core of this definition lies in AC2. Informally, 
the variables in Z should be thought of as describing the "active causal process" from X to ip. These 
are the variables that mediate between X and ip. AC2(a) is reminiscent of the traditional counterfactual 
criterion. However, AC2(a) is more permissive than the traditional criterion; it allows the dependence of 
ip on X to be tested under special structural contingencies, in which the variables W are held constant 
at some setting w'. AC2(b) is an attempt to counteract the "permissiveness" of AC2(a) with regard to 
structural contingencies. Essentially, it ensures that X alone suffices to bring about the change from tp to 
-up; setting W to w' merely eliminates spurious side effects that tend to mask the action of X. 

To understand the role of AC2(b), consider the rock-throwing example again. Looking at the simple 
model, it is easy to see that both Suzy and Billy are causes of the bottle shattering. Taking Z = {ST, BS}, 
consider the structural contingency where Billy doesn't throw (BT = 0). Clearly [ST <— 0,BT <— 0]BS = 
and [ST <— 1,BT <— 0]BS = 1 both hold, so Suzy is a cause of the bottle shattering. A symmetric 
argument shows that Billy is also the cause. 

But now consider the model described in FigurePJ] It is still the case that Suzy is a cause in this model. 
We can take Z = {ST, SH, BS} and again consider the contingency where Billy doesn't throw. However, 
Billy is not a cause of the bottle shattering. For suppose that we now take Z = {BT, BH, BS} and consider 
the contingency where Suzy doesn't throw. Clearly AC2(a) holds, since if Billy doesn't throw (under this 
contingency), then the bottle doesn't shatter. However, AC2(b) does not hold. Since BH G Z, if we set BH 
to (it's original value), then AC2(b) requires that [BT <- 1, ST <- 0, BH <- 0] (BS = 1) hold, but it does 
not. Similar arguments show that no other choice of (Z, W) makes Billy's throw a cause. 

B Proofs 

B.l Proof of Theorem gj] 

First we prove membership in FP Np [ logn l by describing an algorithm in FP Np [ logn l for solving RESP- 
CIRCUIT. The algorithm queries an oracle Ol c for membership in the language L c , defined as follows: 

L c = {(C, X', f, i) : dr(C, X', /') > 

In other words, (C' , X' , /', i) G L c if there exists a set Z of variables of size at most i — 1 such that X' 
is critical for C under the assignment f^. It is easy to see that L c G NP. Indeed, given a set Z of size at 

most i — 1, the check for whether X' is critical for C under can be performed in time linear in the size 
of C. Given input (C, X, f), the algorithm for solving RESP-CIRCUIT performs a binary search on the 
value of dr(C, X, /), each time dividing the range of possible values for dr(C, X, f) by 2 according to the 
answer of Ol c . The number of possible candidates for dr(C,X, f) is the number of variables that appear 
in C, and thus the number of queries to Ol c is at most [log n\ , where n is the size of the input. 



We now prove FP NP t logn l -hardness by a reduction from the problem CLIQUE-SIZE, which is known 
to be FP Np [ lo s™l -complete |Papadimitriou 1984[ IKrentel 19881 |Papadimitriou 1994[ . CLIQUE-SIZE is 



the problem of determining the size of the largest clique of an input graph G. The reduction works as 
follows. Let G = (V, E) be a graph. We start by constructing a circuit Cg, where the variables are the 
nodes in V, and the output of the circuit is 1 iff the set of nodes assigned forms a clique in G. The circuit 
Cq is Cq = A(y w)<£e{V V W). It is easy to see that the value of Cg under an assignment / is 1 iff there 
are edges between all pairs of nodes that are assigned by /. In other words, the set of nodes assigned 
by / forms a clique in G. 

Now let X be a variable that does not appear in Cg- Consider the circuit C = X A Cq, and an 
assignment F that assigns to all variables in V and to X. It is easy to see that the value of C under F is 0, 
and that for an assignment / that assigns X the value 1, C outputs the value of Cg under the assignment / 
restricted to V. We claim that dr(C, X, F) = 1/i > iff the size of the maximal clique in G is \V j — i + 1, 
and dr(C, X, F) = iff there is no clique in G. 

We start with the "if" direction. Let dr(C,X,F) = l/i > 0. Then there exists a set Z C V of 
size i — 1 such that Fg(C) = ^FgufX}^' Since Fg(X) = 0> we a ^ so nave Fg{C) = 0> and thus 
zu{x}(^) = 1- Therefore, the value of Cg under the assignment Fg restricted to V is 1. Thus, the set of 
variables assigned in F? forms a clique in G. The assignment Fg differs from F precisely on the values 
it assigns to variables in Z; thus, the set of variables assigned by F% is V\ Z. We know that \Z\ = i — 1, 
therefore \V \ Z\ = \V\ — i + l. On the other hand, by the definition of the degree of responsibility, for all 
sets Z C V of size j < i — 1 we have Fg(C) = _, -^f U {x} (^)- Thus, the value of Cq under the assignment 
Fa restricted to V is 0. Thus, for all sets Z C V of size j < i — 1, we have that V \ Z is not a clique in 
G. Therefore, the maximal clique in G is of size |V| — i + 1. 

For the "only if" direction, let Y C V of size \V\ — i + 1 be the maximal clique in G. Then the value 
of Cg is 1 under the assignment i^y- Therefore, F,^x ^ u r x y (C) = 1, while F^,^(C) = F(C) = 0. 

Thus, X is critical for C under the assignment Fp<p, and therefore dr(C, X, /) > i. On the other hand, 

since Y is maximal, for all sets Z of size \V\ — j for j < i — 1, we have that Z is not a clique in G, thus 
the value of Cg is under the assignment Fp^g. Therefore, F,p* g)u{X} (^) = = Fp^g(C), and thus X 
is not critical for C under the assignment Fp<g. It follows that dr(C, X, F) < i. Since dr(C, X, /) > i, 
we get that dr(C, X, F) = i. 

If dr(C, X, F) = 0, then for all sets Z C we have F^ u{x} (C) = F^(C) = 0, and thus F^(C G ) = 0. 
Thus, there is no clique in G. For the converse, assume that there is no clique in G. For the other 
direction, assume that there is no clique in G. Then for all Y C V, we have F^^(Cg) = 0, thus 

F (V\Y)U{X}( C ) = f v\y( C ) = °- 11 follows that *( C ' X ' F ) = °" 
B.2 Proof of Theorem 021 

The lower bound follows from the lower bound in Theorem l4.ll For the upper bound, we use the following 
observation made by Eiter and Lukasiewicz: for binary causal models, the condition AC2 can be replaced 
by the following condition (to get an equivalent definition of causality): 

AC2'. There exist a partition (Z,W) of V with X C Z and some setting (x',w') of the variables in 
(X, W) such that if (M, u) \= Z = z* for Z G Z, then 

1. (M, u) \= [X *- x', W <- w']^ip. 



2. (M, u) \= [X *- x, W *- w', Z «- z*]ip. 

That is, for binary causal models it is enough to check that changing the value of W does not falsify ip 
if all other variables keep their original values. Thus, given a partition (Z, W) and a setting (x f , w') we 
can verify that (X = x) is an active cause in polynomial time: both conditions in AC2' are verifiable by 
evaluating a Boolean formula under a given assignment to its variables. Thus checking causality in binary 
models is in NP. Therefore, the following language L' c is also in NP. 

L' c = {((M, u),ip, (X = x), i) : the degree of responsibility of (X = x) 
for ip in the context (M, u) is at least l/i}. 

Indeed, membership of {(M,u),i/j, (X = x),i) in L' c is verifiable in polynomial time similarly to the 
causality check with the addition of measuring the size of witness W, which has to be at most % — 1. The 
algorithm for computing the degree of responsibility of (X = x) for the value of ip in the context (M, u) 
performs a binary search similarly to the same algorithm for Boolean circuits, each time dividing the range 
of possible values by 2 according to the answer of an oracle to the NP language L' c . The number of queries 
is bounded by [logn], where n is the size of the input, thus the problem is in FP NP I logri l. 
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